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ABSTRACT 

Nova  Research,  Inc.  has  recently  developed  a  novel  two-dimensional  imaging  chip  whose  design  is  based  on  properties 
exhibited  by  biological  retinas.  The  “Variable  Acuity”  imager  permits  the  user  to  program  a  unique  spatial  arrangement  of 
“superpixels”  that  may  be  updated  in  real  time.  In  effect,  any  spatial  configuration  of  pixels  in  the  imager  may  be  realized  by 
programming  the  device  in  a  way  that  permits  pixels  to  share  their  individually-collected  photocharge  with  any  or  all  of  their 
neighbors.  Single  and  multiple  “foveal”  configurations  are  possible,  and  these  high  spatial  resolution  regions  may  be 
“flown”  around  the  FPA  at  the  will  of  the  controlling  processor. 

This  device  was  developed  through  the  combined  requirements  of  (a)  covering  a  wide  total  field  of  view  while  (b)  retaining 
the  highest  possible  spatial  resolution  on  the  targets  of  interest  while  at  the  same  time  (c)  operating  at  the  highest  possible 
frame  rate.  Many  thousands  of  frames  per  second  are  possible  with  the  prototype  imager  while  maintaining  high  spatial 
resolution.  The  prototype  device  operates  as  a  visible  imager,  and  Nova  is  pursuing  the  transition  of  this  technology  into  the 
infrared  domain.  This  paper  will  concentrate  on  applications  of  the  technology  and  will  show  some  imagery  collected  with 
the  prototype  system. 

Variable  acuity,  superpixels,  FPA,  programmable,  electronics,  UAV 

1.  INTRODUCTION 

Under  sponsorship  from  the  Air  Force  Research  Laboratory  Munitions  Directorate,  Nova  Research,  Inc.  has  successfully 
demonstrated  the  operation  of  a  novel  focal  plane  array  (FPA)  device  that  has  the  capability  to  be  re-programmed  into  a 
limitless  variety  of  spatial  pixel  configurations.  The  Variable  Acuity  Superpixel  Imager  (VASI)  offers  numerous  new 
capabilities  that  will  open  up  a  variety  of  new  applications  for  systems  designers. 

This  paper  presents  an  overview  of  the  VASI  device  and  provides  some  initial  concepts  for  applications  which  may  be 
realized  through  the  use  of  this  new  imaging  technology. 

2.  OVERVIEW  OF  VARIABLE  ACUITY  DEVICE 

Based  upon  technology  developments  produced  over  the  last  decade,  it  is  now  possible  to  design  a  focal  plane  array  (FPA) 
that  incorporates  a  dynamic,  user-defined  spatial  distribution  of  pixels.  Although  virtually  any  spatial  configuration  (i.e.,  size 
and  location)  of  pixels  could  be  defined,  an  important  first  system  that  has  been  realized  has  been  programmed  to  represent 
the  spatial  configuration  of  the  vertebrate  fovea. 

A  “foveal  FPA”  as  discussed  here  has  the  property  of  higher  spatial  frequency  of  pixel  channels  near  the  “center  of  attention” 
(COA),  with  a  radially- symmetric  spatial  frequency  diminishing  radially  out  from  the  COA.  The  device  as  described  permits 
the  user  to  define  any  desired  spatial  distribution  of  pixels,  and  may  change  this  distribution  at  the  frame  rate,  if  desired.  In 
this  way,  the  COA  may  be  directed  to  move  within  the  total  field  of  view  of  the  FPA  in  order  to  track  objects  of  interest  with 
high  angular  precision,  without  sacrificing  the  ability  to  detect  other  potential  targets  which  may  enter  into  the  sensor’s 
periphery. 
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To  be  demonstrated  throughout  the  remainder  of  this  paper,  every  operational  advantage  required  of  a  tactical  imaging  sensor 
would  be  afforded  through  the  use  of  this  new  class  of  imaging  device: 

•  User-defined  spatial  distribution  of  pixels  permits  high  accuracy  “on  target”,  while  retaining  detection  over  the  entire 
field-of-view  (FOV).  Each  pixel  gets  programmed  into  a  specific  “superpixel”  state,  and  it  will  “know”  how  to  share  its 
charge  with  neighbors  such  that  a  resulting  superpixel  distribution  is  produced. 

•  High  frame  rates  are  possible  by  programming  the  device  into  a  superpixel  state;  pixels  (whether  they  be  “standard 
pixels”  or  “superpixels”)  need  to  deliver  only  one  analog  data  value  per  frame.  This  will  minimize  the  total  number  of 
pixel  values  required  to  be  delivered  off-FPA,  thus  maximizing  effective  frame  rate. 

This  general  methodology  could  be  applied  to  the  use  of  multispectral  detector  arrays  as  well.  These  capabilities  are  now 
possible  through  the  use  of  “READIN”  commands  to  the  FPA,  used  to  program  the  characteristics  of  individual  pixels  in  the 
array.  Figure  2-1  demonstrates  the  concept  of  “FPA  READIN  Programmability”,  resulting  in  a  user-defined  effective 
distribution  of  pixels.  Also  indicated  in  the  figure  is  that  the  readout  state  for  each  pixel  may  be  programmed  individually 
through  the  use  of  the  READIN  command. 
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Figure  2-1  User-supplied  commands  are  “read  in”  to  each  pixel,  resulting  in  unique  superpixel  and  readout  state 

attributes. 


The  READIN  word  multiplexed  into  each  pixel  of  this  VASI  will  be  used  in  each  pixel  to  set  it’s  unique  superpixel  and 
readout  state.  Notice  that  ANY  spatial  configuration  for  the  VASI  would  be  possible;  only  two  such  configurations  are 
shown  in  the  figure.  This  capability  would  also  permit  a  variety  of  spatial  convolutions  to  be  performed,  opening  up  an 
entirely  new  area  of  advanced  on-FPA  image  processing  to  the  technical  community. 

The  superpixel  configuration  of  each  pixel  is  used  to  configure  switch  closures  in  an  on-chip  network  that  interconnects 
neighboring  pixels.  In  effect,  the  combined  integration  capacitance  for  a  given  superpixel  will  be  the  parallel  capacitance  of 
all  connected  unit  cells.  The  combined  photocharge  accumulated  on  the  effective  integration  capacitance  produces  a  voltage 
that  represents  the  signal  on  this  effective  superpixel. 

In  order  to  achieve  a  high  frame  rate,  only  one  value  per  superpixel  need  be  multiplexed  off-chip.  In  addition,  the  device 
permits  every  standard  pixel  to  be  multiplexed  out  (even  though  it  may  be  part  of  a  superpixel)  for  diagnostic  and  other 
imaging  purposes.  In  concept,  as  the  row  select  signal  ripples  down  a  column  of  the  device  during  the  readout  operation, 
only  those  pixels  which  have  been  previously  programmed  to  produce  their  output  signal  will  do  so.  Nova’s  VASI  design 


also  incorporates  a  digital  output  channel  that  produces  a  “read-back”  of  the  currently-programmed  superpixel  state,  such  that 
it  can  be  recorded  with  the  gray-level  imagery  produced  by  the  device.  This  feature  makes  it  possible  for  the  reconstruction 
of  the  superpixel-based  image  data  at  a  later  time.  A  display  processor  with  this  information  and  the  superpixel  gray-level 
values  will  be  able  to  reconstruct  a  display  representing  the  appropriate  spatial  information  in  the  scene. 

Figures  2-2  and  2-3  illustrate  the  advantages  of  the  VASI  readout  design  over  conventional  on-chip  data  reduction  techniques 
of  pixel  dilution  and  region-of-interest  (ROI)  windowing.  With  superpixels,  only  one  output  channel  is  required  to  obtain 
significant  improvements  in  frame  rate  while  not  giving  up  any  scene  information.  And  having  one  analog  output  channel 
rather  than  two,  four,  or  sometimes  even  32  channels  greatly  simplifies  off-chip  A/D  and  re- vectoring  electronics. 


N:1  Dilution 

-  One  out  of  N  pixels  read  out 

-  Fill  factor  reduced  by  (N-l)/N 


MxM  Super-Pixels 

-  One  out  of  N  pixels  read  out  (N=M2) 

-  Fill  factor  remains  unchanged 


□  Read  out  pixel 

□  Skipped  pixel 
—  Pixel  boundary 


Figure  2-2  Dilution  modes  throw  away  pixel  data  in  exchange  for  higher  frame  rates  to  maintain  total  FOV  (left), 
while  a  comparable  superpixel  configuration  maintains  both  FOV  and  fill  factor  since  all  skipped  pixels  share 

photocharge  with  read  out  neighbors. 


Windowing 


Super-Pixels 


-  Sub-frame  regions  read  out  successively 

-  IFOV  reduced  to  increase  frame  rate 

-  Size  and  position  usually  limited  by  the 

number  of  address  bits 

-  Rectangular  only 


-  High  acuity  and  low  acuity  regions  read  out 

-  IFOV  remains  unchanged  with  increased 

frame  rate 

-  Any  shape  and  location  permitted 
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Figure  2-3  Windowing  modes  reduce  instantaneous  FOV  (IFOV)  while  increasing  frame  rate  (left),  while  a 
comparable  superpixel  configuration  can  read  out  both  the  high-resolution  ROI  as  well  as  the  low  resolution 
peripheral  pixel  regions  with  little  additional  clocking  overhead. 


Nova  has  produced  an  operating  prototype  version  of  a  VASI  device  operating  with  a  monolithic  visible  phototransistor  in 
each  unit  cell  (30  micron  unit  cell  pitch),  as  shown  in  Figure  2-4.  Initial  imaging  and  test  results  presented  in  this  paper  were 
produced  from  this  test  device.  As  of  the  writing  of  this  paper,  a  larger  128  x  128  pixel  device  is  in  fabrication. 


Figure  2-4  Nova’s  prototype  32  x  32  VASI  visible  imager  is  monolithic  in  construction  with  a  phototransistor  in  each 

unit  cell. 


3. 


PRELIMINARY  IMAGING  AND  TEST  RESULTS 


Prior  to  imaging  the  device,  initial  tests  verified  the  correct  operation  of  the  unit  cell  programmability.  Figure  3-1 
demonstrates  the  results  of  programming  a  uniform  4x4  superpixel  array  configuration  in  which  the  behavior  of  the  readout 
multiplexor  is  properly  controlled  by  the  superpixel  state.  In  this  case,  every  other  column  is  skipped  in  the  horizontal 
readout  direction  and  every  other  row  is  skipped  in  the  vertical  readout  direction,  resulting  in  a  4-to- 1  data  reduction.  Scope 
traces  0-5  in  the  figure  represent  the  six  input  clocks  required  by  the  device.  Traces  0-2  consist  of  master  clock,  integration 
and  reset  clocks  for  readout  operation;  traces  3-5  consist  of  superpixel  programming  clock,  programming  sync  and  serial 
programming  commands.  The  programming  clocks  may  operate  completely  asynchronously  to  the  readout  clocks — during 
these  tests  the  master  clock  ran  at  2.5  MHz  while  the  superpixel  programming  clock  ran  at  10  MHz.  This  minimized  frame 
overhead  time  which  could  be  used  instead  for  pixel  integration. 


(a)  (b) 


Figure  3-1  Oscilloscope  waveforms  of  one  readout  frame  in  both  non-superpixel  configuration  (a)  and  4x4  superpixel 
configuration  (b)  showing  analog  pixel  output  trace  (top)  and  digital  output  trace,  and  the  input  clocks  (0-5).  Note 
that  in  superpixel  configuration  only  one-fourth  of  the  pixel  values  are  multiplexed  off-chip,  allowing  for  an  increased 

frame  rate. 


As  mentioned  above,  the  readout  has  a  superpixel  disable  mode  in  which  all  pixel  values  are  read  out  regardless  of  the 
programmed  superpixel  configuration.  In  this  mode,  the  chip’s  digital  output  can  be  used  as  a  means  of  verifying  each 
individual  pixel’s  programmed  superpixel  state.  This  information  can  also  be  used  to  reconstruct  the  images  when  the 
readout  is  put  back  into  superpixel  enable  mode,  allowing  display  electronics  to  work  autonomously  with  respect  to 
command  control  electronics.  For  example,  if  the  control  system  is  reprogramming  the  superpixel  state  every  100  frames, 
then  the  first  frame  of  each  100  frame  sequence  should  be  acquired  while  superpixel  mode  is  disabled  in  order  to  store  a 
complete  spatial  configuration  profile  from  the  digital  output  pad  (digital  output  is  not  valid  when  in  superpixel  enable 
mode). 

For  the  sake  of  imaging  tests  performed  in  the  lab,  the  readout  was  kept  in  superpixel  disable  mode  so  that  a  consistent  32x32 
array  format  could  be  acquired  by  the  display  system.  Figure  3-2  shows  two  examples  of  foveated  superpixel  configurations 
realized  using  lxl,  2x2  and  4x4  pixel  sizes.  Such  configurations  can  be  reprogrammed  at  the  frame  rate  in  order  to  “fly”  the 
high-acuity  regions  within  the  FOV  in  real-time  according  to  where  the  COA  directs  them. 

Figure  3-3  shows  the  digital  output  that  results  from  a  center-foveated  superpixel  configuration  depicted  in  Figure  3 -2(a). 
The  “Up”  image  data  appears  during  the  first  half-clock  of  a  pixel  period,  and  the  “Left”  image  data  appears  during  the 
second  half-clock.  Used  together,  a  frame  of  superpixel  data  can  be  fully  reconstructed.  “Up”  and  “Left”  denote 
connectivity  between  an  individual  pixel  and  its  upward  and  leftward  nearest  neighbor. 


Figure  3-2  Two  superpixel  configurations  are  shown  for  the  32  x  32  device.  Panel  (a)  has  a  central  fovea,  panel  (b) 

has  located  the  foveal  region  in  the  upper  left  corner. 


Figure  3-3  These  images  represent  the  “Up”  and  “Left”  bits  used  to  program  the  configuration  shown  in 
Figure  3-l(a).  These  bit  values  are  produced  from  a  digital  output  pad  that  the  device  has  such  that  the  instantaneous 
spatial  configuration  of  the  device  may  be  recorded  with  image  data. 


The  above  images  were  generated  using  static  (pre-determined)  programming  clock  timing  from  the  test  system  which  did 
not  allow  for  real-time  COA  movement.  A  simple  single-CPLD  based  control  system  was  subsequently  developed  that 
allowed  a  user  to  move  an  8x8  high-acuity  region  within  a  4x4  superpixel  field  by  dragging  the  computer  mouse  in  a  console 
window.  The  mouse  coordinate  was  transmitted  as  an  8 -bit  address  across  the  parallel  port  where  the  CPLD  used  it  to 
position  the  high-acuity  region  by  shifting  the  superpixel  programming  command  sequence.  Figure  3-4  shows  snapshots 
from  live  image  sequences  taken  while  operating  in  standard  configuration  (no  superpixels)  and  in  this  superpixel 
configuration.  Spatial  noise  evident  in  panel  (b)  is  a  result  of  the  high  contrast  display  setting  acting  on  some  test  set  output 
cable  noise  pickup. 


Figure  3-4  Two  images  from  a  real-time  video  sequence  demonstrating  non-superpixel  configuration  (a)  and  a 
programmed  8x8  high-acuity  region  within  a  field  of  4x4  superpixels  (b). 


Six  frames  were  extracted  from  a  large  sequence  of  contiguous  image  frames  when  the  VASI  imager  was  operating  in  its 
standard  “non-superpixel”  mode  and  are  shown  in  Figure  3-5 (a).  A  focused  laser  spot  was  imaged  onto  the  array  and  the 
collected  imagery  shows  the  expected  high  resolution  pixellation. 


Figure  3-5(a)  Six  frames  taken  from  a  sequence  with  the  variable  acuity  sensor  operating  in  its  high  resolution  mode 
(i.e.,  no  super-pixellation).  A  focused  moving  laser  spot  is  imaged  onto  the  array. 


In  this  mode,  the  32  x  32  array  may  operate  with  a  frame  rate  of  up  to  approximately  1000  Hz.  When  the  device  was 
programmed  to  have  a  central  foveal  spatial  characteristic  as  depicted  in  the  image  shown  in  Figure  3-2(a),  a  large  set  of 
contiguous  frames  was  collected,  six  frames  from  which  are  shown  in  Figure  3-5 (b).  Notice  how  the  spatial  extent  of  the 
extended  object  is  captured  in  a  high  spatial  resolution  mode  when  the  object  is  near  the  center  of  the  field  of  view,  and  how 
many  fewer  pixels  are  used  to  represent  the  image  when  the  object  is  in  the  periphery  of  the  field  of  view.  This  super-pixel 
mode  of  operation  makes  use  of  approximately  6.4  times  fewer  pixel  channels  than  the  full-resolution  example  above  because 
of  super-pixellation.  In  theory,  this  reduction  in  the  number  of  effective  pixel  channels  permits  this  imagery  to  be  produced 
at  a  frame  rate  of  approximately  6.4  kHz  (ignoring  frame  overhead). 

In  addition,  the  temporal  noise  produced  by  a  superpixel  will  be  reduced  as  compared  to  “individual  pixels”  by  virtue  of  the 
root-sum-square  collection  of  detector  currents  contributed  by  all  of  the  individual  pixels  that  contribute  charge  to  the 
superpixel’s  effective  charge  well.  Many  future  applications  that  require  the  ultimate  in  low-noise  performance  may  make 
use  of  this  approach  to  reducing  effective  pixel  temporal  noise. 


Figure  3-5(b)  Six  frames  taken  of  a  focused  moving  laser  spot  with  the  variable  acuity  sensor  was  operating  with  a 

high  spatial  resolution  foveal  region  in  the  center  of  the  array. 


4.  BIOLOGICALLY-INSPIRED  DESIGN  ATTRIBUTES 

Many  examples  in  nature  have  demonstrated  the  use  of  multiple  acuity  foveal  (i.e.,  regions  of  high  spatial  acuity) 
architectures,  as  reported  by  Wolpert1 .  “The  human  eye  has  one  foveal  region.  The  fovea  measures  approximately  400  i  in 
diameter  and  has  a  photoreceptor  density  of  approximately  150,000  photoreceptors/mm2.  Some  birds  of  prey  have  a  density 
of  photoreceptors  of  up  to  1,000,000/mm2,  some  7x  greater  than  those  of  humans.  Greater  resolution  and  a  means  to  process 
the  information  without  exceeding  data  rates  and  overloading  the  processor  (brain)  is  a  goal  of  almost  every  detector 
manufacturer. 

“Some  shore  birds  also  have  multiple  foveae,  one  that  is  rectangular  and  oriented  horizontally  like  some  of  the  animals  of  the 
African  savannah,  conforming  to  the  sky/water  and  sky/land  horizon.  They  also  have  another  central  circular  fovea,  straight 
ahead,  that  is  used  for  tracking.  Optimizing  the  foveae  for  the  detection  process  can  enhance  and  augment  detection  for  some 
applications. 

“Some  animals  have  multiple  foveae  like  the  anableps  fish.  This  fish  has  an  eye  with  two  different  focal  lengths  and  two 
foveae.  The  long  focal  length  portion  of  the  lens  focuses  to  one  fovea  for  viewing  in  air  the  other,  short  focal  length,  focuses 
below  water.  Taking  advantage  of  this  technique  allows  lenses  to  be  designed  for  multiple  media  applications. 


Figure  4-1  The  Anableps  eye  has  two  focal  lengths  and  two  foveal  regions2 

“Most  birds  of  prey  also  have  two  foveae.  Like  search  &  track  systems  that  employ  a  short  focal  length  optical  system  for 
acquisition,  and  a  long  focal  length  system  for  tracking,  the  eye  is  oblong  and  equipped  with  two  foveae.  The  lens  is  rotated 
in  the  socket  to  switch  between  search  and  track  modes.  It  is  necessary  to  understand  the  "hand-off"  process  that  is  between 
search  and  track  modes  that  allows  these  birds  of  prey  to  continue  tracking  after  detection  without  loosing  its  prey.  This 
hand-off  process  is  critical  in  modern  search  and  track  systems.” 

Future  applications  of  VASI  technology  are  only  limited  by  the  imagination  and  inspiration  of  systems  designers,  and  may  be 
spawned  by  further  investigations  into  how  biological  systems  have  evolved  such  systems. 


1  Wolpert,  H.D.,  “Bio  Mimicking”,  private  communication,  2002. 

2  http://ebiomedia.com/gall/eyes/4eye.html 


5. 


APPLICATIONS  OF  VASI  TECHNOLOGY 


High  Frame  Rate  Operation 

Superpixel-mode  operation  is  useful  in  applications  requiring  high  effective  frame  rate  without  sacrificing  total  field-of-view 
or  the  spatial  resolution  on  objects  of  interest  within  the  scene.  Reducing  the  total  number  of  pixels  per  frame  which  must  be 
read  off  of  the  imaging  device  will  reduce  the  effective  frame  period,  thus  increasing  effective  frame  rate.  For  example,  the 
superpixel  configuration  as  shown  in  Figure  3-2(a)  can  operate  at  a  frame  rate  of  approximately  6.4  times  faster  than  that  for 
multiplexing  the  entire  high-resolution  frame.  In  this  mode,  Nova’s  prototype  visible  VASI  sensor  as  shown  in  Figure  2-4 
operates  at  a  frame  rate  of  approximately  6.4  kHz. 

Possible  high  frame  rate  applications  would  include  high  speed  image  motion  capture/freezing,  remote  sensing  of  object 
closing  velocities,  passive  depth  measurement  at  high  speed  using  at  least  two  sensors  and  particle  fragment  trajectory 
measurements. 

Tracking  Systems 

Motion  sensing  techniques  may  be  used  to  direct  the  VASI  device  to  “track”  moving  objects  by  placing  high-resolution 
pixels  over  the  moving  target,  leaving  the  rest  of  the  image  area  in  a  low-resolution  spatial  condition.  By  comparing  the 
center  location  of  the  “foveal”  region  to  the  center  of  the  total  field-of-view,  a  tracking  error  signal  is  produced  that  may  be 
used  to  servo  the  sensor  platform  so  as  to  reduce  the  pointing  error.  While  covering  a  very  wide  total  field-of-view,  high 
resolution  target  imagery  is  maintained  while  keeping  the  target  near  the  center  of  the  image.  Significant  processing 
resources,  if  applied  to  the  center  of  the  field  of  view,  would  thereby  retain  a  high  degree  of  relevance  on  the  target. 

Vibration/Platform  Stabilization 

External  sensors  (vibrometers,  accelerometers,  etc.)  attached  to  the  sensor  platform  may  be  utilized  to  produce  two- 
dimensional  error  signals  which  may  be  used  to  generate  the  requisite  foveal  position  control  to  the  VASI  imager.  Image 
data  contained  in  the  foveal  region  of  the  sensor,  having  “tracked”  platform  motion,  would  approach  a  “pixel  stabilized” 
condition  useful  in  improving  a  system’s  ability  to  identify  objects  of  interest. 

Nova  has  recently  produced  an  updated  version  of  the  visible  VASI  imager  that  has  a  VASI  imaging  region  128  x  128  pixels 
in  size,  with  proprietary  optical-based  “angle  rate  sensing”  circuitry  on-chip  that  may  be  used  to  produce  the  stabilization 
signals  required  for  this  mode  of  operation.  A  range  of  “Unmanned  Air  Vehicle”  (UAV)  applications  exist  for  such  a  sensor 
configuration. 

Template  Matching  Operations 

Pre-determined  object  shapes  may  be  used  to  program  one  or  more  “foveal”  regions  of  the  VASI  imager.  When  the  desired- 
shape  object  is  imaged  onto  the  sensor  and  becomes  spatially  aligned  with  these  spatially-formed  foveal  regions,  a  high 
degree  of  signal  correlation  will  exist  in  the  output  of  these  foveal  regions.  A  following  image  processing  system  could 
evaluate  this  correlation  condition  so  as  to  produce  a  “probability  of  match”.  Since  the  spatial  configuration  of  the  fovea 
may  be  re-programmed  at  the  frame  rate  of  the  sensor,  this  would  provide  a  convenient  means  for  producing  such  two- 
dimensional  correlations  in  real  time. 
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